BIGTOP-3921: Add ZSTD Codec Support for hadoop#1095
BIGTOP-3921: Add ZSTD Codec Support for hadoop#1095rzuo wants to merge 2 commits intoapache:masterfrom
Conversation
|
We need fix for toolchain too to support this on all supported distros/platforms. |
|
Is the Zstandard licensed under the one we can bundle it by default? |
In bigtop_toolchain/manifests/packages.pp, libzstd-devel has been included, so anything else need to modify? Thanks |
By my understanding, this fix only make the hadoop native library compiled with pre-installed ZSTD binary, no ZSTD (GPL-2.0 license) code included in this repository. Thanks |
zsdt's license is BSD (https://github.com/facebook/zstd/blob/dev/LICENSE). |
aha, you are correct, i miss understood that zstd is GPL licensed. |
Oops. I forgot about BIGTOP-3535. |
We are publishing pre built packages for users' convenience. We can not link it against pre-built library if it is distributed under GPL. Zstandard seems to be dual-licensed under BSD and GPLv2. I'm not confident about which is applied to pre-built libzstd.so redistributed by OS distros. |
| [ -f /usr/lib/libzstd.so ] && BUNDLE_ZSTD="-Dbundle.zstd=true -Dzstd.lib=/usr/lib" | ||
| [ -f /usr/lib64/libzstd.so ] && BUNDLE_ZSTD="-Dbundle.zstd=true -Dzstd.lib=/usr/lib64" | ||
| [ -f /usr/lib/${HOSTTYPE}-linux-gnu/libzstd.so ] && BUNDLE_ZSTD="-Dbundle.zstd=true -Dzstd.lib=/usr/lib/${HOSTTYPE}-linux-gnu" |
There was a problem hiding this comment.
Can we expect all supported distro/platform have libzstd.so? We can add package dependency on libzstd to make it certain that the required library is installed to runtime environment if so.
There was a problem hiding this comment.
emmm, I only have CentOS 7.4 at hand.
Can someone help to verify other distros/platforms?
There was a problem hiding this comment.
emmm, I only have CentOS 7.4 at hand. Can someone help to verify other distros/platforms?
Assume your development environment is x86 64/amd64-based.
Please try to test it in various Distros of Bigtop docker images for x86 64/amd64 first.
(https://hub.docker.com/r/bigtop/slaves/tags)
There was a problem hiding this comment.
Ok, I'll paste the result later, thanks.
There was a problem hiding this comment.
With bigtop docker images for x86_64, got following result:
- centos-7
… rpm package, so that no system zstd library dependency is needed.
I could completely understand your concerns for What do you think of it? @iwasakims |
|
Spark bundles pre-built binary of zstd-jni (containing zstd as part of it). I'm concerning about the license of libzstd binary distributed by OS distros here. |
3 participants
Description of PR
Add ZSTD codec compiled to hadoop native library by default.
How was this patch tested?
For code changes: